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Abstract 

Critical appraisals of the literature may help to increase reproducibility in neuroscience. Systematic 
reviews and meta-analyses are tools for neuroscientists to critically evaluate a large amount of knowledge 
in the research field. These methods involve individually simple decisions, which may become complex 
when considering the whole process. Strategies to organize the planning and implementation of the 
protocols minimize the workload. Here, we prepared a compact guide to assist neuroscientists willing to 


perform a systematic review and meta-analysis of the literature in neuroscience. 
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Introduction 

Critical appraisals of comprehensive 
literature may help neuroscientists to identify 
more consistent and reproducible findings in the 
research area. The exponential accumulation of 
scientific literature prompted methodologies to 
filter and synthesize evidence (1). Systematic 
reviews (SR) are transparent and unbiased 
methods to identify, obtain, filter, appraise, and 
synthesize studies from the literature to answer 
a research question (2). Meta-analysis (MAN) 
is the name given to the pool of statistical 
methods used to combine quantitative results of 
different studies into a single one (3, 4). 
Combining SR with MAN (SRMAN) makes a 
method to obtain a quantitative synthesis of 
unbiased information from the literature (5). 
The SRMAN has been considered the highest 
level of evidence guiding decision making in 
Medicine (6, 7), despite the controversies 
beyond the scope of this text (4). In any case, 
the synthesis of evidence applied to 
neuroscience may be helpful to conceive a novel 
hypothesis or identify gaps in the knowledge 
about a given subject (8, 9, 10). Moreover, 
critical appraisal of evidence may optimize 
future research, minimizing the waste of 
scientific efforts (11). 

Synthesizing evidence often involves labor 
from teams of reviewers (scientists, 
methodologists, librarians, statisticians, others) 
and meticulous preparation to avoid mistakes 
and biases (12, 13). SRMAN comprises a 
sequence of individually simple actions. 


DOI: 10.31885.jrn.2.2021.1669 


Complexity emerges when considering the 
whole sequence of steps required to perform 
SRMAN, while maintaining the quality 
standards. To guide reviewers through the best 
practices, the scientific community created 
guidelines such as “Preferred Reporting Items 
for Systematic Reviews and Meta-Analyses” 
(PRISMA) and PRISMA-extensions (14, 15, 
16). PRISMA provides guidelines to publish 
results of reviews transparently (14). Guidelines 
also encourage systematic reviewers to prepare, 
register, and disseminate protocols before 
performing a SRMAN (15, 16). 

Tools to organize the processes help 
minimize the workload associated with the 
planning and execution of a SRMAN. For 
SRMANs of clinical and observational studies, 
Cochrane collaboration offers __ training, 
instructions, and tools to perform the complete 
process _—_(https://cccrg.cochrane.org/). For 
SRMANs of nonclinical or preclinical studies in 
laboratory animals, free resources may be found 
at the Systematic Review Center for Laboratory 
Animal Experimentation (SYRCLE, 
https://www.radboudume.nl/en/research/depart 
ments/health-evidence/systematic-review-center 


-for-laboratory-animal-experimentation) or 
Collaborative Approach to Meta-Analysis and 


Review of Animal Data of Experimental Studies 
(CAMARADES, 

https://www.ed.ac.uk/clinical-brain-sciences/res 
earch/camarades/about-camarades). Nonclinical 
in vivo, ex vivo, and in vitro studies are 


prevalent in neuroscience, making the 
frameworks by SYRCLE or CAMARADES 
more suitable than Cochrane’s framework to the 
synthesis of evidence in this field. 

The scientific literature offers excellent 
manuals providing step-by-step instructions on 
how to perform a SRMAN in different fields of 
research (6, 8, 10, 13, 17, 18). For example, 
Vesterinen (10) made a detailed guide on 
performing SRMAN of animal studies in the 
biomedical field. On the other hand, Muka (13) 
created a simplified guide with 24 steps to 
perform SRMAN synthesized medical studies. 
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Here, we combined Vesterinen (10) and Muka 
(13) approaches to provide a_ simplified, 
compact guide for neuroscientists wishing to 
perform a SRMAN for the first time. For this, 
we organized a flowchart with five consecutive 
phases of an overview of the complete process 
involved ina SRMAN (Figure 1): 1- elaboration 
of a review question; 2- elaboration of a review 
protocol; 3-protocol preregistration; 4- protocol 
implementation; 5- review publication. The 
following text presents a theoretical background 
for each phase with a brief description and 
examples taken from neuroscience. 
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Figure 1. Phases of a systematic review and meta-analysis. The process of systematic review and meta-analysis split into five 
consecutive phases: 1- review question elaboration; 2- protocol elaboration; 3- protocol preregistration, 4- protocol execution; 5- 


review publication. 


Phase 1: Elaboration of a Review Question 

In our opinion, the first, the most crucial 
phase of a SRMAN is the definition of the 
research question, which will guide all other 
steps of the review (19). Neuroscientists are 
ideal personnel to create a relevant research 
question to the literature in neuroscience. Before 
preparing the research question, neuroscientists 
should scan the literature to check if the same 
topic has been previously reviewed. When a 
strong rationale for a review has been found, 


neuroscientists should elaborate on the review 
question. Neuroscientists, especially the novice 
users of systematic reviews, may benefit from 
the aid of librarians or other methodologists in 
elaborating a searchable review question. 
Searchable research questions may be 
elaborated using mnemonic tools, helping 
reviewers remember the components of a 
well-elaborated, direct, and relevant question to 
the science field (20, 21, 22). Cochrane 
collaboration endorses the PICO (P — Patient or 


population or problem; I — Intervention; C — 
Comparison; O — Outcome) tool when 
reviewing controlled, randomized, clinical trials 
(23). Even though neuroscience studies are 
mostly nonclinical, PICO might be helpful for 
creating review questions in this field. PICO 
(Table 1) will be particularly useful to review 
studies investigating effects of new treatments 
(e.g., natural or synthetic compounds) or other 


types of interventions (e.g., toxins, 
environmental manipulations, etc.) on 
quantitative outcomes (behavior, 


neurochemistry, neuroanatomy, etc.) obtained 
from experimental subjects (human volunteers 
or laboratory animals are common experimental 
subjects in neuroscience). 

Experimental designs in neuroscience are 
more variable and flexible than in clinical 
research (26, 27). Therefore, variations of PICO 
such as PICOC (PICO plus Context) or PICOT 
(PICO plus Timeframe) may be helpful (22). 
Besides PICO and variations, other tools that are 
potentially useful when reviewing neuroscience 
literature include SPICE (S — Setting; P — 
Population; I — Intervention; C — Comparison; E 
— Evaluation) or SPIDER (S — Sample or 
population of interest; P — Phenomenon of 
Interest; D — Design; E — Evaluation; R — 
Research type). In addition, PICOC, PICOT, 
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SPICE or SPIDER may be handy when specific 
types of research, such as in vitro, in vivo, ex 
vivo, or brain imaging is the review's main 
interest. These last tools may also be suitable 
when reviewing studies reporting qualitative 
outcomes (e.g., cell shape or position or color) 
and contexts influencing the outcomes (e.g., the 
timing of treatment). 

Hence, using mnemonic tools, a complex 
review question may be stated in more simple, 
direct terms (e.g., "has the intervention, I, 
changed the outcome, O, in the population, P, 
compared to control treatment, C?"). The 
direction of the expected change of the outcome 
(decrease or increase) stated in the review 
question depends on the theoretical background 
defined by the authors of the question. Based on 
the experience with the literature in the field, 
neuroscientists may want to know, for example, 
"have antidepressants decreased the immobility 
time of rats measured in the forced swimming 
test as compared to the vehicle?" (28). Other 
neuroscientists may ask, "have antidepressants 
increased hippocampal neurogenesis in 
laboratory rats as compared to the vehicle?" 
(29). Moreover, mnemonic tools will help 
reviewers plan the searches of publications, 
select relevant studies to answer the review 
question, and analyze the outcomes. 


Table 1: PICO tool: categories, definitions and examples. 


Category Definition 


Population 


from which the outcomes were measured. 
Intervention 


measured. 
Comparison 


(or Control) 
to the population. 


Population of interest, i.e., biological units 
receiving the intervention or control treatment 


The experimental treatment was applied to the 


population from which the outcomes were 


A reference to the effects of the intervention 
on the population. Standard treatment applied 


Examples 


In vivo studies: volunteers, rats, mice, laboratory animals, and 
in vitro studies: primary cultures, immortalized cell cultures 
etc. 

Chemical treatments (prototypical compounds, new 
compounds, toxins, so on); environmental manipulations 
(enriched or impoverished housing for laboratory animals, 
conditioned media for culture, so on). 

Vehicle compared to the chemical treatments; standard 
environmental conditions (standard housing for laboratory 
animals, standard media compared for culture, so on) 
compared to alternative conditions (enriched or impoverished 
housing for laboratory animals, conditioned media for culture, 


5 SO on). 


Outcome Measures taken from the population of 


interest. 


Phase 2: Elaboration of a Review Protocol 

In phase 2, a protocol is set to annotate the 
suitable methods to answer the research 
question elaborated in phase 1. A protocol of a 
SRMAN may comprise as many as fifty 
different methodological and analytical 
decisions, which may lead to different answers 
to the review question. The extension 
PRISMA-P (15) was created to help systematic 
reviewers elaborate a protocol of SRMAN 
following good practices 


Behaviors registered in behavioral tests, protein concentration 
in samples, number of neurons or glial cells in brain slides, 
and so on. 


(http://www. prisma-statement.org/Extensions/Pr 
otocols). Decisions range from identifying 
potential reviewers and collaborators to the 
results' plans. Templates have been built to 
guide reviewers through the long series of 
decisions in a protocol for SRMAN. Cochrane 
collaboration (30) and SYRCLE (28) offer 
templates to prepare SRMAN protocols of 
studies on humans and animals, respectively. 
Advantages of using a template include 
preparing a complete plan facilitating the 


registration of the protocol in public platforms. 
Librarians, statisticians, or other methodologists 
(e.g., experienced reviewers) are essential 
collaborators for elaborating a SRMAN 
protocol, especially for the first time. 

Besides  authors' names, addresses, 
affiliations, review title, and the review question 
described in terms of the mnemonic tool, a 
protocol for SRMAN should contain a brief 
introduction to the research subject. SRMAN 
protocols should describe the strategies to obtain 
the publications from the literature and filter 
relevant studies to answer the review question. 
A complete protocol of SRMAN provides the 
plans for the assessments of internal and 
external validities, qualitative and quantitative 
analysis, and the appraisal of the impact of bias 
on the synthesized evidence (31). A detailed 
protocol indicates the number of independent 
reviewers involved in the different steps of the 
review, how analysis of 
agreements/disagreements between reviewers 
and conciliation of disagreements will be 
performed. Due to the massive amount of 
information produced during the different steps 
of a SRMAN, especially in critical activities 
such as screening processes and data extraction, 
ideally, two independent reviewers and a third 
one for conciliation should be involved in each 
step. 

In the following topics, we provide a short 
description with examples of the items typical 
of a SRMAN protocol. A hypothetical review 
question, elaborated using a PICO tool (e.g., 
"has the intervention, I, changed the outcome, 
O, in the population, P, compared to control 
treatment, C?"), is the framework for the 
examples presented below. 
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Search strategy: This item consists of a 
detailed description of how publications will be 
retrieved from bibliographic bases. Good search 
strategies are sensitive and specific, usually 
resulting from small, iterative probes in the 
bibliographic bases, called pilot studies (18). 
Pilot studies prevent inadequate retrievals or 
errors, leading to studies capable of answering 
the review question. On the other hand, broad 
and poorly targeted searches can lead to 
superfluous articles and a loss of time for the 
reviewers (13). A search strategy should contain 
the list of search terms (keywords), Boolean 
operators and syntaxes used in the searches 
performed in bibliographic databases. Search 
terms may be chosen with the aid of the same 
mnemonic tool used to elaborate the review 
question (Table 2). Boolean operators (e.g., 
AND, OR) are used to combine the search terms 
(Table 2). For example, if the targeted 
population is "laboratory animals", SYRCLE 
offers filters for a comprehensive search of the 
literature in Medline and Embase (28, 32). 
Librarians may provide assistance for advanced 
searches in different bibliographic bases and 
reference management. Additionally, 
bibliographic databases platforms offer updated 
tutorials on their websites. For example, see 
examples of protocols presenting search 
strategies in neuroscience by Ramos-Hryb (33) 
and Bolzan and Lino de Oliveira (34). 
Warnings: 1- searches in different bibliographic 
bases combined will provide a more 
comprehensive review than one; 2- different 
search engines operate differently; 3- access to 
some bibliographic databases require payment 
or institutional login. 


Table 2. Description of a generic search strategy elaborated to obtain relevant publications to answer a hypothetical review question 


created using the PICO tool: 


Search Search terms Retrievals (n) 
#1 Terms related to (population P or synonyms n, 
#2 Terms related to (intervention I or synonyms) n; 
#3 Terms related to (control C or synonyms) ny 
#4 Terms related to (outcome O or synonyms) ny 
Combination of terms used in individual searches, i.e., (population P or Dhico 
#5 synonyms) and (intervention I or synonyms) and (control C or synonyms) 


and (outcome O or synonyms) 


The search terms field should contain the string of search terms combined by Boolean operators. Every search engine has specific 
tules to do advanced searches, e.g., some of them require Boolean operators in capital letters, others require specific commands to 
enter queries, and so on. Therefore, training in search engines is advisable. Reviewers should decide on the adequacy of the search 
terms to avoid spurious searches (e.g., are keywords related to comparison or controls necessary to obtain relevant studies?). Pilot 
studies to determine the best search strategy are advisable. The number of publications retrieved in a given search (n) helps to 


inform about the precision and validity of the searches. 


Screening or selection strategy: In this 
item, reviewers should describe how the 


relevant studies will be selected among the 
publications retrieved from bibliographic bases. 


The selection or screening process aims to 
identify the most relevant publications to answer 
the review question(s) (35). The screening 
process is easier and unbiased based on the 
inclusion and _ exclusion criteria defined 
beforehand (36). The output of this process will 
provide lists of references excluded from the 
review with explicit reasons and a list of studies 
included in the review. Included studies are the 
more relevant publications to answer the review 
question(s) and will be analyzed in the SRMAN 
(36). The selection strategy includes planning 
screening phases (title and/or abstract, and/or 
full text), number of independent reviewers, and 
conciliators involved. Additional decisions 
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include prioritizing exclusion criteria in the 
screening phases (e.g., excluding duplicates or 
reviews in the first phase, i.e., title and/or 
abstract). The mnemonic tool used to elaborate 
the review question is helpful to guide the 
choice of the eligibility criteria (Table 3). For 
example, criteria related to population (e.g., 
include if rat; exclude if mice), interventions 
(e.g., include if tricyclic antidepressants; 
exclude if all antidepressants except tricyclic), 
and outcomes are often used in reviews planned 
using the PICO tool. See examples of protocols 
presenting screening strategies in the field of 
neuroscience by Hohls (37), Ramos-Hryb (33), 
and Bolzan and Lino de Oliveira (34). 


Table 3. Description of generic inclusion and exclusion criteria to a screening strategy of studies relevant to answer a hypothetical 


review question created using the PICO tool. 


Category Inclusion criteria Exclusion criteria 
Population Studies using population P Studies using populations a, b or c 


Intervention 


Comparison (control) 


Outcome Studies reporting outcome O 


Type of study 
Type of publication 


Studies using intervention I 


Studies using control C 


In vivo studies 


Original studies 


Studies using interventions a, b or c 
Studies using controls a, b or c 
Studies reporting outcomes a, b orc 
In vitro or ex vivo studies 


reviews, systematic reviews, meta-analysis 


Reviewers should decide about the adequacy of other relevant categories of eligibility criteria (data, language, experimental design). 
Exclusion criteria may be more loosely defined than inclusion criteria, e.g., "studies using any population except P" or "studies using 
any intervention except I" or "studies reporting any comparison/control except C" or "studies reporting any outcome except O" 


Assessment of internal validity: In this 
item, authors should describe the approaches 
reviewers will use to appraise internal validity, 
i.e., how well the studies included in the review 
(primary studies) or the review itself were 
conducted and reported. There are various tools 
to assist the reviewer in assessing the 
methodological quality of the included studies 
(65, 66, 67). A tool such as RoB-SYRCLE 
(26), adapted from the RoB tool by Cochrane 
(38) to animal experimentation, should be more 
suitable for risk of bias assessment in the field 
of neuroscience. Although not mandatory in a 
protocol of SRMAN, the assessment of internal 
validity of the SRMAN itself may be planned. A 
tool like the AMSTAR 2, applicable to evaluate 
the internal quality of systematic reviews, 
including randomized or non-randomized 
studies of health interventions (39), may be 
helpful for the assessment of SRMAN in the 
field of neuroscience. The number of 
independent reviewers and conciliators involved 
in the process should be indicated. The protocol 
should also provide an analysis of agreements 
between reviewers (e.g., Cohen Kappa) (40). 
See an example of a protocol for planning 
assessment of internal validity of studies in the 
field of neuroscience at Galindo (41). 


Assessment of external validity: This item 
should describe the approaches planned to 
appraise the external validity of primary studies, 
i.e., generalizability, consistency, reproducibility 
of findings. These evaluations depend on the 
analysis of qualitative information extracted 
from primary studies, such as types of 
experimental designs, population, exposure, or 
interventions and results. The authors should 
describe the methods for obtaining these 
qualitative data, including the number of 
independent reviewers and conciliators involved 
in the process, the localization of the 
information to be extracted (e.g., texts, graphs, 
tables), and how the authors of the included 
studies will be contacted to provide missing or 
additional data. The protocol should also 
provide an analysis of agreements between 
reviewers (e.g., Cohen Kappa) (40). Protocols 
should contain the list of all the qualitative 
information reviewers decided to extract from 
the primary studies to evaluate the external 
validity of the literature in their research fields. 
Each neuroscientific subfield, or individual 
review, should decide regarding the relevance of 
qualitative aspects of the primary studies to 
evaluate external validity (e.g., species, strain, 
age, sex of experimental animals or subjects; 


type, doses, route of administration, and 
regimen of drug and vehicle administration; 
type of measures taken of assays or trials, so 
on). See examples of protocols planning 
assessment of external validity of studies in 
neuroscience at Van Praag (42) and Eckert (43). 

Effect sizes estimation: Systematic 
reviewers often elaborate questions, which the 
calculation of effect sizes can answer (e.g., how 
large is the effect of the intervention, I, on the 
outcome, O, of the population, P, as compared to 
the control group, C?). Effect sizes are statistics 
to estimate the difference between groups or the 
strength of the relationship between variables 
affecting an outcome (dependent variable) (44). 
In the fictitious example at the beginning of this 
item, the calculation of mean differences could 
indicate how large the difference between the 
values of the outcome O, measured in the 
population P, under the influence of treatment I 
compared to the control C. In other words, the 
mean difference signifies "the effect of the 
treatment I on the outcome O". The reviewer 
should specify the type (dichotomous? 
continuous?) and the units of measurement of 
the primary or secondary outcomes according to 
the protocol to the prespecified review question. 
Primary outcomes are those essential to answer 
the review question, while secondary ones are 
optional. The methods for obtaining the data, 
number of independent reviewers involved in 
data extraction, solution of discrepancies or 
missing information, and tabulation of the data 
should be planned. The protocol should also 
analyze agreements between reviewers (c.g., 
Cohen Kappa) (40). Protocols should also 
explain how effect sizes will be calculated (e.g., 
mean difference, odds ratios) using quantitative 
data extracted from primary studies (e.g., mean, 
standard deviation, sample sizes, number of 
comparison, p, F, or t values, correlation 
indexes, or any other measure, manipulation, or 
transformation of the data, so on). The choice of 
the calculations of effect sizes depends on the 
type of targeted outcome (dichotomous? 
continuous?), type of study design (controlled? 
unpaired or paired design?) and comparison of 
interest (males compared to females? treatment 
compared to control? before treatment 
compared to after treatment?). Effect sizes are 
often estimated and reported, with indicators of 
uncertainty (confidence intervals, standard error, 
standard deviation). For a more detailed 
description of effect size calculations, see 
Durlak (44). See examples of protocols planning 
effect sizes estimations of studies in the field of 
neuroscience at Galindo (41), Soflau (45), and 
Husain (46) in the “effect measure”. 
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Meta-analysis: Effect sizes calculated from 
individual studies may be synthesized or 
combined in a global MAN or stratified in 
subgroups. A protocol should specify, per 
outcome, if a MAN is planned, how the decision 
of doing it or not will be taken, how estimated 
effect sizes of primary studies will be combined, 
and the indicators of the uncertainty of the 
combined effect size (confidence intervals, 
standard error, standard deviation, 
heterogeneity) (47). The number of studies 
available may be a criterion to decide whether a 
MAN is feasible or not. Theoretically, two 
studies are sufficient to calculate a combined 
effect size. However, as with other statistical 
methods, a meta-analysis may provide spurious 
results with small sample sizes (48). Thus, if 
sample size calculations are planned, the 
protocol should inform how calculations will be 
performed (e.g., power analysis). MAN's power 
analysis may be performed using the Metapower 
package in R (49). In the case of a feasible 
MAN, the reviewer must pre-specify the 
statistical model of analysis (e.g., random 
effects or fixed effect model); the statistical 
methods used to evaluate heterogeneity (e.g., I, 
Q); subgroup analyses; sensitivity analyses and 
evaluation of publication bias. The decision on a 
suitable statistical analysis model depends on 
the degree of heterogeneity anticipated for the 
MAN (50). Random effect models are often 
eligible to MAN in research fields like 
neuroscience, in which a variety of study 
designs are often employed to investigate a 
similar research question. A high degree of 
heterogeneity may also justify the planning of 
the subgroup analyses (51). Relevant subgroups 
explaining the variability of data are dependent 
on the research field, and authors may stratify 
the MAN into subgroups according to the 
categories selected for external validity 
assessment (e.g., features of population, 
intervention, control or outcomes). For example, 
in animal studies, the species, strain, age or sex 
of laboratory animals are expected to affect the 
results of a study. If the assessment of 
publication bias is planned, it should be also 
included in the protocol. Publication bias, 
prevalent in basic research (52), may be 
assessed using the trim-and-fill analysis and 
funnel plotting method (53, 54). The complexity 
of the study design may sometimes require 
calculations beyond classical MAN methods 
such as network MAN, for example, Dias and 
Caldwell (55). See examples of protocols 
describing plans for MAN in the field of 
neuroscience at Pozza (56), Galindo (41) and 
Santos (57) in the “strategy for data synthesis”. 


Dissemination plans: Planning for 
disseminating the results comprises part of the 
review process and should be prespecified, even 
vaguely, in the protocol to accommodate the 
expectations of all the authors involved. 
Systematic reviews often involve large groups 
of reviewers and authors should decide how all 
contributors will get the credit. Some 
periodicals permit the use of group names (e.g., 
The NPQIP Collaborative group, (58)), while 
others may request the list with all contributors. 
It is up to the authors to identify the appropriate 
media to make the results available to the 
appropriate public. See examples of protocols 
describing dissemination plans in the field of 
neuroscience at Alvarez-Bueno (59) or Bolzan 
and Lino de Oliveira (34). 


Phase 3: Protocol Registration 

Preregistering the protocol is recommended 
to avoid overlaps and superfluous efforts and to 
encourage the rigor and transparency of studies 
(13, 15, 27). Protocol registration before starting 
to perform the SRMAN may help reviewers 
stick to the plan, as a long list of decisions needs 
to be made over the process, reducing the 
incidence of biased reviews. Furthermore, 
transparent dissemination of the protocols may 
attract collaborators to join the group of 
reviewers, which may be beneficial to perform 
the plan, especially when handling many 
publications and studies. 

PROSPERO is a public platform specialized 
in registrations of protocols for SRMAN of 
studies relevant to human health in humans or 
laboratory animals 
(https://www.crd.york.ac.uk/prospero/). 
Currently, rapid reviews and umbrella reviews, 
not scoping reviews, are also eligible for 
PROSPERO registration. The Open Science 
Framework (OSF, https://osf.io/) is also a public 
and free of charge platform used to deposit 
scientific protocols of any study, including 
SRMAN protocols. Protocols may also be 
published in scientific journals with peer review 
(e.g., 33, 34). 


Phase 4: Protocol Implementation 

Phase 4 comprises the implementation of all 
decisions included in the protocol created (phase 
2) and preregistered (phase 3) beforehand. 
Although time-consuming, protocol 
implementation may be less complex than 
protocol elaboration. In our experience, most of 
the actions involved in searching publications 
and selecting studies using pre-established rules 
can be quickly learned by inexperienced 
reviewers. The extraction of data (quantitative 
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and qualitative) from included studies is the 
most time-consuming step of a SRMAN and can 
be challenging, especially for inexperienced 
systematic reviewers (11). Applying the 
pre-established analytical choices _—_ are 
technically simpler than planning a MAN due to 
the availability of intuitive tools and software 
(e.g., 60). Perhaps procedures to guarantee data 
integrity and tracking the process are the most 
considerable challenges in this phase; thus, 
inexperienced systematic reviewers will benefit 
from interaction with more experienced 
collaborators. Ideally, deviation of the protocol 
during implementation should be annotated to 
be presented in the final report. A public 
platform like OSF (https://osf.io/) may be a 
valuable resource for reviewers to keep track of 
all documents created in the process of a 
SRMAN. 

There are many different free or commercial 
resources available to implement the processes 
in a SRMAN; degrees of automatization vary 
from step to step. For example, the metagear 
package for R is a free resource facilitating 
screening, data extraction, and MAN (61). 
Packages for R such as metagear (61) or pacman 
(62) may be used to calculate the agreement 
between reviewers (Cohen Kappa) (40) 
whenever two independent reviewers performed 
an activity (e.g., screening process, data 
extraction, and so on). In the following text, we 
make suggestions of selected free resources to 
implement each step of a SRMAN in any field 
of nonclinical research, including 
neurosciences: 

Searching publications, deduplication: 
The "advanced search" menu of search engines 
(e.g., Pubmed, Web of Science, Embase, 
Scopus) are often more appropriate for the type 
of search required in a systematic review than 
the simple search menu. Recovering documents 
from different virtual bases requires reference 
management software for deduplication (63). 
Mendeley 
(https://www.mendeley.com/download-referenc 


e-manager/), Zotero (https://www.zotero.org/) 
and Rayyan (64) are examples of free managers. 


CAMARADES developed a web application for 
deduplication (Automated Systematic Search 
Deduplication Tool, ASySD, 
https://www.ed.ac.uk/clinical-brain-sciences/res 
earch/camarades/tools-resources). The number 
of publications obtained in the searches 
performed in every bibliographic database, 
before and after deduplication, should be 
annotated in the final report. 

Screening relevant studies: Reference 
management software is handy to apply the 


selection criteria (63). Mendeley 
(https://www.mendeley.com/download-referenc 
e-manager/) and Rayyan (64) can be used to 
classify references in a library 
semi-automatically. Rayyan supports the initial 
screening of abstracts and titles using a 
semi-automation process while incorporating a 
high level of usability (63, 64, 66). Other free 
resources are ASReview (https://asreview.nl/), 
Parsival (https://parsif.al/), Rvtools (tools for 
evidence synthesis in R, https://revtools.net/), 
Sysrev (https://sysrev.com/), “Screenatron and 
Systematic Review Accelerator’ (66), SyRF 
(https://syrf.org.uk/ , CAMARADES). All these 
software allow for multiple reviewers’ 
assessment facilitating independent judgments 
and _conciliations. Results of screening 
processes (number of excluded studies, reasons 
of exclusion, number of included studies) 
should be annotated to be presented in the final 
report. 

Data extraction for internal or external 
validity assessment and_ effect size 
estimations: Assessments of internal or external 
validities require information extracted from the 
text of primary studies. Various software 
applications support extracting text from PDF, 
including free trials of proprietary software 
(PDF readers). The reference managers 
themselves can be helpful to obtain 
bibliographic information (e.g., authors' name, 
year, journal, so on). CAMARADES are 
developing approaches to automate data 
extraction to facilitate the application of the 
RoB-SYRCLE tool (67). Reviewers interested 
in applying to AMSTAR 2 may find a helpful 
checklist available at: 
https://amstar.ca/index.php. Online tools such as 
Colandr (https://colandrcommunity.com/, 
Colandr Community), Systematic Review Data 
Repository (SRDR) or SyRF 
(https://syrf.org.uk/, CAMARADES) _ enable 
efficient form for annotating, building, sharing, 
and data management (68). Although most of 
the above-mentioned online tools cannot 
automatically extract data from primary 
publications, they allow for multiple reviewers' 
assessments facilitating independent judgments 
and conciliations. Tabulation of extracted 
information provides a database that will 
facilitate MAN, data sharing and the final 
report. 

Data extraction for the effect sizes 
estimation, meta-analysis calculations and 
plots: In the field of neuroscience, most of the 
quantitative data from primary studies are in 
graphs and/or tables. Although time-consuming, 
manual measurements of scaling or bar sizes, 
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line extensions, and other graphical attributes 
are accessible using the digital ruler (e.g., free 
Adobe Reader). Some knowledge of the 
programming language allows using the R 
metaDigitise package to extract descriptive 
statistics such as means, standard deviations, 
correlations automatically from different chart 
types (box, scatter, histograms) (69). Tabulation 
of the extracted information provides a database 
that may facilitate MAN, data sharing and final 
report. Software specific for MAN will provide 
the values of effect sizes with confidence 
intervals of primary studies, the combined effect 
size along with measures of uncertainty (e.g., 
confidence intervals, standard error), 
heterogeneity and plots [70, 71]. Tools such as 
Meta-Essentials 
(https://www.erim.eur.nl/research-support/meta- 
essentials/ (60), RevMan (72) or OpenMEE (73) 
can be used by reviewers unfamiliar with 
statistical software or programming language in 
conducting MAN (74, 75). Reviewers more 
familiar with coding can benefit from software 
such as meta or Metafor packages in R (74); 
Python or OpenMeta_ [Analyst] (60). 
Information about software and packages 
selected for data extraction and MAN should be 
annotated to the final report. 


Phase 5: Review Publication 

A transparent report may help readers 
appraise the reliability and validity of the results 
of the SRMAN and other neuroscientists to 
reproduce the study. PRISMA Statement has 
developed several extensions to facilitate 
reporting different types or aspects of systematic 
reviews 
(http://www. prisma-statement.org/Extensions/). 
Reviewers may adopt different strategies to 
publish, share and disseminate results of the 
review, including scientific events, workshops 
or online platforms and peer-reviewed journals 
(several available). In peer-reviewed journals, 
unless otherwise specified, the report of 
SRMAN may be organized like other types of 
publications, including an introduction to the 
subject, methods, results, and discussion (76, 
77, 78, 79, 80). The report of a SRMAN may 
also be a part of a narrative review (29). Some 
journals request that articles report a SRMAN to 
submit a completed PRISMA checklist, 
available at www.prisma-statement.org or 
journal websites to facilitate the peer-review 
process. 

The Introduction of an article reporting a 
SRMAN contains the theoretical background, 
hypothesis, and the review question stated using 
the terms of the mnemonic tool. Methods may 


be briefly presented, primarily when the 
reviewers can provide the reference or 
identification number of the preregistered 
protocol (e.g., PROSPERO number or a link of 
OSF). The Methods section could be divided 
into subsections describing the search strategy, 
screening strategy, data extraction, quality 
assessments and analysis. Detailed descriptions 
of the search and screening strategies are pivotal 
to replicating the review but are frequently too 
extensive to be presented in an article. Short 
versions of the search and screening strategies 
may be presented in the Methods along with the 
reference or link to access the detailed strategies 
(e.g., a link of OSF). Reporting the date of the 
searches is always crucial because the number 
of publications retrieved may change over time. 
The methods used to extract the qualitative and 
quantitative data from the primary studies 
should be presented. Approaches selected to 
assess internal and external validity should be 
mentioned and described. Important: 
calculations and interpretations of effect sizes 
should be explicitly stated because they have an 
impact on the discussion of data. A description 
of the analytical choices of the quantitative 
analysis, with or without a MAN, should be 
presented in the Methods section, even when the 
preregistered protocol is cited. 

The Results section may be divided into 
subsections describing the output of the search 
and screening strategies, quality assessments of 
the included studies and, finally, meta-analysis. 
The number of studies retrieved in each 
bibliographic database, excluded by exclusion 
criterion, fulfilling inclusion criteria, included in 
the qualitative analysis, and included in the 
quantitative analysis should be reported and also 
in a PRISMA flowchart (36) presented as a 
Figure in the manuscript. Results of the RoB 
assessment may be presented as tables or charts. 
For example, the plot for RoB-Syrcle may be 
performed using the Robvis R package (81). 
Qualities of the individual studies may be 
reported in the text or tables. Global or 
subgroups meta-analysis results are often 
presented in a Forest plot (82), enabling the 
appraisal of individual and combined effect 
sizes in a single figure. Scatter plots are 
standard, especially when meta-regressions are 
reported (83). Funnel plots (84, 85) are 
especially recommended when publication bias 
assessment is reported. A link (e.g., link of 
OSF) containing files with raw data will help 
readers appraise the results' quality. 
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The Discussion involves the examination of 
the internal and external validities of the 
publications included in the review, which is 
valuable for the appraisal of the quality of the 
data available in the research field. Results of 
the MAN should be discussed in terms of 
direction, magnitude, the significance of 
individual and combined effect sizes. The 
direction of the effect size, i.e., positive or 
negative values, usually indicate evidence, 
respectively, in favor of or against an 
intervention or treatment or exposure of interest 
(warning: positive or negative signals of effect 
size have no absolute meaning depending on the 
type of calculation performed, e.g., subtracting 
low value means from high-value means will 
give a negative effect size). In neuroscience, the 
benchmarks to interpret the magnitude and 
significance of the effect sizes are yet to be 
established. In other fields of research, effect 
sizes have been classified from small to very 
large (86, 87), while confidence intervals 
excluding null may be considered "statistically 
significant" (88). 

Publication bias, prevalent in basic research, 
often inflates combined effect sizes, distorting 
the evidence's appraisal (89, 90). The impact of 
heterogeneity on the combined effect size 
should be addressed. In the field of 
neuroscience, features of the population (e.g., 
species, strain, age, sex), intervention (dose of 
the drug, via of administration), types of control 
groups or types of outcomes may have an 
impact on results of the studies. Authors may 
discuss how stable the effect sizes were across 
subgroups. In the conclusions, authors may 
summarize how reliable the review results are 
according to the internal and external validities 
assessments. Additionally, authors may 
elaborate on how heterogeneity and publication 
bias may be affecting the conclusions of the 
review. Finally, limitations of the review process 
might be disclosed. 
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